Fractional‐order operators on nonsmooth domains

The fractional Laplacian $(-\Delta )^a$, $a\in(0,1)$, and its generalizations to variable-coefficient $2a$-order pseudodifferential operators $P$, are studied in $L_q$-Sobolev spaces of Bessel-potential type $H^s_q$. For a bounded open set $\Omega \subset \mathbb R^n$, consider the homogeneous Dirichlet problem: $Pu =f$ $, $u=0$ $ R^n\setminus\Omega $. We find regularity solutions determine exact domain $D_{a,s,q}$ (the space $u$ with $f\in H_q^s(\overline\Omega )$) cases where has limited smoothness $C^{1+\tau }$, for $2a<\tau <\infty $0\le s<\tau -2a$. Earlier, domains were determined smooth $\Omega$ by second author, was found low-order H\"older $\tau =1$ Ros-Oton Serra. $H_q^s$-results obtained now when new, even )^a$. In detail, identified as $a$-transmission $H_q^{a(s+2a)}(\overline\Omega )$, exhibiting estimates terms $\operatorname{dist}(x,\partial\Omega )^a$ near boundary. result required new development methods handle nonsmooth coordinate changes operators, which have not been available before; this constitutes another main contribution paper.